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Goals of this Presentation 


Provide overview of biofeedback 
Discuss potential applications 
Use examples to show costs and benefits 
Discuss future directions and implications 


Biofeedback Overview 


Biofeedback: measurement, display, 
analysis, modification, manipulation, 
and response of physiological signals 

Using biological indicators to index 
sentiment/emotion 


Biofeedback Overview 


Feedback loop possible where subsequent 
signals depend on prior states 

Emotional states not stable 

Transient 

Volatile 

Subject to manipulation 
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Why Biofeedback? 


Current control schemes 

Provide one dimension of input 
Map player intent to onscreen action 
Ignore other aspects of cognition 
Ignore player sentiment 



http://www.ps3controller.org/ 


http://www.geeky-gadgets.com/useful-gadgets-the-usb-wireless-mini-keyboard-and-mouse/ 






Why Biofeedback? 


What about player sentiment? 

Adding emotional input incorporates 
new (and previously ignored) 
dimension of player input 

Tailor more immersive, dynamic, and 
calibrated game experience 



http://www.ps3controller.org/ 


http://www.geeky-gadgets.com/useful-gadgets-the-usb-wireless-mini-keyboard-and-mouse/ 
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http://www.ps3controller.org/ 


http://www.geeky-gadgets.com/useful-gadgets-the-usb-wireless-mini-keyboard-and-mouse/ 











Emotion 


Subjective, internal state induced by 
response to (usually) external events 

Vector 

Magnitude (arousal) 

Direction (valence) 


Valence 


Arousal 


Angry 

Energetic 

Jubilant 

Afraid 

Engaged 


Happy 

Sad 

Content 

Bored 


Fatigued 


Passive 

Relaxed 


Adapted from Lang (1995) 





Physiological Signals 


Heart rate 

SCL (skin conductance level) 
Facial expressions 
Eye movements 

EEGs (Electroencephalography) 

Others (pupil dilation, body 
temperature, posture, etc.) 


Heartrate 


Beat to beat interval of blood flow 
Measure baseline rate and deltas over time 





















Heart Rate 


Index of arousal 
Cheap 

Easy to measure 
Familiar 

Fourier transform to 
get valence? 


Prone to movement 
artifacts 

Delayed onset to 
stimuli 

Difficult to determine 
valence 


Voltage (mv) 


SCL 


Electrical resistance of the skin 
Chart waveform of arousal over time 
Get responsive and anticipatory spikes 



http://www.tbiomed.eom/content/2/1 /11 /figure/F5?highres=y 














SCL 


Index of arousal 
Tonic/Phasic responses 
Minimal lag to stimuli 
Cheap 

Robust to movement 

Lots of measurement 
sites 


Difficult to associate 
eliciting events 

Difficult to determine 
valence 

Range is variable 
across subjects 



Facial Expressions 


Record movement of facial muscles 
Classify emotion (both valence and arousal) 
Can be done remotely or via EMG 



Anger 



Surprise 


Disgust 


Sadness Happiness 


Fear 



http://www.colour-journal.Org/2007/l/2/07102article.htm 



Facial Expressions 


Index of valence 


Can be intrusive 


Index of arousal 

Measures 

instantaneous 

responses 


Expensive 

(at the moment) 

Subject to bias 

Requires training or a 
black box 


Eye Movements 


Remote (or mounted) cameras measure 
reflectivity off of pupils 

Record where eyes are looking in real-time 

Get saccades (movements) and fixations 
















Eye Movements 


Index of attention 

Rudimentary index of 
thought 

Index of arousal (with 
pupil dilation) 

Unique 

Reliable 


Very expensive 

Requires extensive 
analysis 

Can be intrusive-^ 
lead to subject biasing 
eye movements 


Measure electrical potentials of the brain 
Primarily time-based signals 
Coarse measures of location 
Get frequency spectra and spike latency 



http://www.riversideonline.eom/health_reference/Test-Procedure/MY00296.cfm 
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http://bindcenter.eu/?page_id=12 


http://www.frontiersin.Org/human_neuroscience/10.3389/neuro.09.005.2009/full 


http://news.tigerdirect.eom/2008/02/28/put-on-your-thinking-cap-a-gaming-helmet-can-read-minds/ 















Index of arousal 

Index of valence 

Rudimentary insight 
into thought 


Very expensive 
Very intrusive 
Very noisy 
Difficult to validate 


Others 


Pupil Dilation - arousal 
Body temperature - arousal 
Body posture - valence 

Couple with pupil dilation to get frustration 
Lots of stuff we haven’t thought about 


Potential Applications 


Passive viewing of biofeedback data 

Modify game experience based upon 
player sentiment/emotion/internal state 

L4D director with biofeedback 
Adaptive realtime difficulty adjustment 
Detect and respond to disengaged players 

Determine optimal arousal patterns 

Can manipulate gameplay to induce 









Potential Applications 


Physiological data as direct input 

Tie health to arousal 
In-game prompts tied to emotional state 
NPCs respond dynamically 
Required valence/arousal to proceed 


Potential Applications 


Matchmaking/Profiling 

Spectate competitive matches 

Multiplayer Mechanics 
Detect teammate in trouble 
Earn points for eliciting responses 

Playtesting 


Current Experiments 


Modification of Al Director in Left 4 Dead 2 
Addition of physiological input to Alien Swarm 
Eye movements as active controls in Portal 2 






Alien sw/'M 


p0rTAL2 


Current Experiments 


Passive viewing of physiological inputs 

Implications for multi-player 

Playtesting Applications 









Modification of Al Director 


Director creates dynamic, variable 
experience in Left 4 Dead series 

Modifies enemy spawns, health and weapon 
placement, boss appearances, etc. 

In-game encounters determined by 
estimated arousal level 


Modification of Al Director 


Will replacing estimated arousal with actual 
arousal create a more enjoyable experience? 

Can we determine optimal arousal patterns? 


Director Algorithm 


Represent Survivor intensity as single value 
Increase it in response to in-game trauma 
Decay intensity to zero over time 
Create peaks and valleys 


Desired Population 
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Current Hardware Solution 





DeciSeconds 







Analysis of SCL Data 


Categorize game events 
Record survey responses 

Enjoyment, frustration, etc. 

Quantify waveform 

Spike frequency, size of range, average lag, etc. 

Data-mine (correlation, regression, 
frequency analysis, PCA, etc.) 
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3. Overall, I found today's experience enjoyable. 

1 2 3 4 5 

(strongly disagree) 

4. Overall, today's experience was frustrating. 

1 2 3 4 5 

(strongly disagree) 

5. Overall, today's play session was challenging. 

1 2 3 4 5 

(strongly disagree) 


6 7 8 9 

(strongly agree) 


6 7 8 9 

(strongly agree) 


8 9 

(strongly agree) 
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Survey ratings 


How enjoyable was the 
playtesting session? 



0 % 100 % 

Biofeedback contribution 


Survey ratings 


How challenging was the 
playtesting session today? 

9 - 

8 - 



0% 100% 

Biofeedback contribution 
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Component 2 

























Std Spike 

Mean SCR Spike Amp Min Spike 
SCLs nSCR Amplitude Amp Range Amp 

Question4-0.10 -0.09 0.00 -0.07 -0.08 

Questions -0.09 0.25 0.15 0.22 -0.07 

Question6-0.39 -0.03 0.02 0.01 -0.13 


Max 

Spike Mean Spike 
Amp AreaSum 


-0.07 0.07 %Average excitement 


0.22 -0.22 %Average frustration 


0.00 


0.06 


%Average challenge factor 


Results 


Measured arousal produces greater 
enjoyment than estimated arousal 

Have rudimentary insight into events 
which elicit enjoyment 

Progress on optimal arousal patterns 


Experiment Summary 


Physiological signals are viable inputs 
More work needed to ‘quantify’ enjoyment 
How well can we shape the arousal curve? 
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Alien Swarm + Biological Input 


Top-down, team-based action shooter 
Create mod with time-based constraint 

Kill 100 enemies in 240 seconds 
Timer indexed to arousal (SCL) 

Highly aroused-> timer speeds up 
Relax^ timer reverts to baseline 


Alien Swarm + Biological Input 


Can you create a compelling gameplay 
experience using physiological signals 
as direct input? 

What kind of problems will arise? 

Feedback loop? 

Possible manipulations of signal? 



Problems 


Positive feedback loop exists 

Increase in arousal leads to increase in 
arousal... 

Decay factor helps 

Clarity of relationship between arousal 
and in-game events not always clear 



Experiment Summary 


Novel gameplay experiences possible 
Experience qualitatively different 

Aware of both gameplay and emotional response 
LOTS of work required to tweak algorithm 




Play Portal 2 With Your Eyes 


Puzzle-based FPS 

Traditional control schemes use single 
control to shift viewpoint AND crosshair 

Decouple viewing and aiming 
Use hand to move 
Use eyes to aim 


Play Portal 2 With Your Eyes 


Is it enjoyable to use your eyes to aim? 

How do you change gameplay if you add 
more degrees of freedom to aiming? 

Since the eyes move faster than the wrist 
is speed of movement correlated with 
enjoyment? 


Portal 2 Eyetracker Algorithm 


Use eyetracker to extract eye’s X,Y position 
Feed those coordinates into game engine 
Redraw cross-hair at current eye position 
Update at 60 Hz 







Experiment Summary 


Eyes are viable aiming controllers 
Decoupling aiming/viewpoint is a plus 
Interesting question of how to use blinks? 
Best suited to more action-oriented games 
Consumer-grade eye trackers are far away 


Multiplayer 


Show representations of other player 
emotional state? 

Is it engaging to view vital signs of 
teammates/opponents? 

Is it a useful game mechanic? 

Detect distress? 
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Multiplayer Summary 


Most enjoyable thing we’ve done so far 

High sense of satisfaction when opponents spike 

Entertaining to view teammates response 

Not useful (yet) 


Playtesting Applications 


Create more objective responses 

Lots of biases in current playtesting procedures 

Quantify responses 

Encourages rapid iteration on player state 


Overall Summary 


Adding physiological signals opens up 
new dimensions of gameplay 

Novel control schemes worth exploring 

Consumer-grade devices to track both 
valence and emotion are needed 


Future Directions 


Matchmaking on physiological profiles 
Experimentation with gameplay mechanics 
Quantify optimal arousal patterns 
Investigate other hardware platforms 
Incorporate techniques into playtesting 
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